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Campbell (1969) argued for the interrupted 
time-series experiment as a useful methodology for testing 
intervention effects in the social sciences. The validity of the 
statistical hypothesis testing of time-series, is, however, dependent 
upon the proper identification of the underlying stochastic nature of 
the data. Several' types of model misidentif ications are examined for 
some commonly encountered models in the social sciences! Analytic 
expressions for actual Type I error and power probabilities are 
derived when the mathematics is tractable; simulation techniques are 
adopted for the remainder of the cases. Eesults indicate that model 
misidentif ication leads to severe perturbations of the nominal 
probabilities. (Author) 
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Campbell (1969) argued for the interrupted time-series experiment 
as a useful methodological scheme for testing intervention effects 
in the social sciences. The validity of the statistical hypothesis 
testing^of time-series is, however, dependent upon the proper 
identification of the underlying stochastic nature of the data. 
Several types of model misidentif ications are examined for some 
cotmnonly encountered raodels in the social sciences. Analytic ex- 
pressions for actual Type I error and power probabilities are derived 
when the mathematics is tractable; simulation techniques are adopted 
for ^he remainder of the cases. Results indicate that model mis- 
identification leads to severe perturbations of the nominal probabilities. 
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THE CONSEQUENCES OF MODEL MIS IDENTIFICATION IN THE 
INTERRUPTED TIME-SERIES EXPERIMENT ^ 



William L. Padia 
Office of Program E'.valuation and Research 
California Department o£ Education 
721 Capitol Mall 
Sacramento, California 95814 

The interrupted time-series experiment is a useful methodological scheme 
for. testing intervention effects In the social sciences. The wide range of 
application, spurred by Campbell (1969), both in the experimental and quasi- 
experimental settings, has produced a number of interesting designs for tefiiting 
complex interventions. 

The design and analysis settingK are well explicated by Glass, Willson and 
Gottman (1975) and a user-oriented computer program by Bower, Padia and Glass 
(1974) is available to simplify the analysis. Certainly one of the most trouble- 
some and amorphous aspects of time-series methodology is In area of model identi- 
fication. With the high degree of ambiguity present in the model Identification 
stage, it is necessary to know the results of mis-v"lassif ication errors. The 
social scientist must be aware of the severity and direction of these errors- 

The stochastic properties of time-series following the Box and Jenkins 
(1970) formulation, and the importance of these properties as related to the 
significance testing of intervention effects are considered below. The salient 
problem area in regard to statistical hypothesis testing is the proper identi- 
fication of the underlying stochastic process of the time-series. Improper 
model identification yields spurious Type I error and power estimates, the 
nature of which requires systematic examination. 

1 A paper presented at the annual meeting of the American Educational Research 
Association in New York, April, 1977. 
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The most common types of model niisidentif lea tions are based on common 
time-series models observed In social and behavioral settings, Padia (1975): 
white noise processes in undifferenced data; white noise processes in first- 
differenced data; first-order autoregressive processes; and, integrated moving 
average processes, Misidentif ication is defined to encompass the following 
three areas: 

1. Underf itting one model to another such as identifying a second- 
order moving average process as a first-order moving average 
process. 

2. Misfitting one model to another such as identifying a first- 
order autoregressive process as a first-order moving average 

, process. 

3. Improper specification of d . The difference parameter may be 
underestimated as in identifying an integrated moving average 
process as a moving average process in the zeroth difference. 
Similarly, d may be overestimated as in identifying a first- 
order autoregressive process as a first-order autoregressive 
process in the first difference. 

Combinations of these three areas are also considered. Analytic expressions 
for actual error probabilities under the various misidentif icatlons are derived 
when the mathematics is tractable; Monte-Carlo simulation techniques are adopted 
for the remainder of the cases. The results are valid for series in which the 
number of observations is large': analytic solutions are accurate to order l/n 
and simulation results are based on series of length n=122. Detailed analytic 
solutions and simulation procedures found in Padia (1975). 

The salient point with regard to the various misidentifications is that 
the mis-specification of d^ is the dominant influence and, for the most part. 



controls the direction and magnitude of discrepancies from the nominal Type I 
error or the nominal power. Failure to identify d properly leads to spurious 
error probabilities and the direction and magnitude is predictable, in general^ 
irrespective of the stochastic properties or parameter values of the models. 
Proper specification of d is imperative in any statistical significance testing 
procedure involving time-iseries data. 

Several conclusions may be drawn based on a summary of the results with 
some extrapolations where the regularity of the results justifies generalization. 
All conclusions refer to model misidentif ications for certain parameter values 
that are likely to be observed under a particular model misidentif ication . 

Underf its 

Underfits produce Type I error rates which are functions of the magnitude 
and sign of the autoregressive and moving-average parameters. The effects are 
mixed in that some combinations of parameters produce actual Type I probabilities 
less than the nominal, while other combinations produce rates greater than the 
nominal* The results are valid regardless of the value of the difference para- 
meter d, as long as there is agreement with 6 for the misidentif ied model and 
the true model. For the most part, the deviations from the nominal level are 
not trivial and care should be taken to avoid an underf it. Overfitting is never 
a problem since the additional overf it parameters are zero in the true model. 

Misfits 

Misfits, like underfits, produce true Type I error rates which are functions 
of the magnitude and sign of the true and misidentif ied models. The effects 
are mixed and the results are identical for various values of d^, as long as d^ 
in the assumed model agrees with d_ in the true model. 



d Underesttmations 

The most spectacular effects on Type I error occur when a process which 
Is non-stationary in level or non-stationary in Level and slope is assumed to 
be stationary at d := 0, The perturbed Type I error probabilities are 0.80 to 
0,90, quite independent of the stochastic properties of the misidentif ied 
models. The probablistic ''war.derings*' of a series non-stationary -in level 
nearly guarantees the existence of a significant intervention effect under a 
true null hypothesis. Indeed, the effects are so severe that the actual 
rates are very nearly identical for all three nominal levels of 0,10, 0,05 
and 0-01- 

With a series non-stationary in level and slope which has been identified 
as non-stationary in level only, the situation is quite different. In the case 
of a white noise under fit of a white noise process, the nominal o< overestimates 
the actual c?\, contrary to the d = 1 underfits, Onc^ the effects of a series 
non-stationary in level are accounted for (by first differencing), the^various 
models and parameter values are free to exert their influence on the nominal 
Type I error probabilities. The deviations from nominal remain serious with 
the actual less than nominal oC in some cases , and greater in other cases. 

The effects of underestimating d[ within a given model are not additive, 
i.e., actual Type I probabilities for ad=0tod=2 underfit are not the sum 
ofad=Otod^=l underfit and ad[=ltod = 2 underfit. 

d^ Overes tlmatlons 

Overestimating d^ is not as serious as underestimating d . Overdif ferencing 

a d.= 0 series once produces mild discrepancies In the white noise cases with 
actual ti^ less than The ief feet 6^^^^^^^^ 1 to d « 0 overestimate^ 
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for models other than white noise, generally results in actual cK less than 
nominal > with the different model parameters exerting limited influence. 

Overdlfferencing a d[ =« 1 process yields actual p<. levels greater than 
nominal levels in the white noise case and, for the most part, in models 
other than white noise. The type of stochastic process which is overdif fer^enced 
and the values of the parameters have some effect as to the direction and magni~ 
tude of the discrepancies. 

The effects of overdlfferencing are not additive, i.e., actual Type I error 
probabiUties obtained by overdlfferencing a d « 0 process twice are not the 
sum of thti probabilities associated with overdlfferencing a d = 0 and d = 1 
process. 

Combinations 

When a combination of a misfit or underflt with a mis-specification of d 
occurs, the resulting effect on Type I error is a function of the d mis-speci- 
ficaticu. With d underestimates (true d = 1) the effects of a misfit or under- 
fit are compliBtely obfuscated by the overwhelming influence of the d^ mis^specifi 
cation. The probabilities are similar to the d^ underestlTnates considered alone, 
i.e., in the 0.80 to 0.90 range. With d underestimates (assumed d= 1, true 
d = 2) the departares from nominal are not as spectacular, the effect of the 
underestimate is not as dominate (although, in general. actualoK is less than 
nominale^ as the underestimate predicts) and the model and model parameters 
have some influence. 

As expected, in the case of a d overestimate (assumed d = I, true d_ = 0 
or assumed d = 2, true d^ = 1 ) in combination with a misfit or underfit, the 
dominant influence is the mis-specification of d^. The actualc^ is close to 
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the nominalc<, although the models and their parameters have some influence 
as to the direction and magnitude of the discrepancy. 

Since the predominant influence on the system is the mis-specification of 
d, the effects on Type I error on the combined misidentification model are not 

the sum of the separate effects, but controlled rather tightly by the mis-speci- 

fication of d. 

Power Conclusions 

Since power and Type I error form an inextricable relationship, the effects 
of model misidentif ica tibn on power are closely related to the effects of model 
misidentif ication on Type I error. In those cases where d^ is properly identified 
the nominal power may overestimate or underestimate the actual power; the extent 
and direction of the deviation from nominal is a function of both the particular, 
misfit or underfit and the values of the model parameters. The departures from 
nominal are moderate to severe depending upon the particular case under study. 

In those situations where d is underestimated (true d = 1) , the result is 
that an intervention effect will be detected regardless of its magnitude, the 
true power being near to 0.9 for the three levels considered. The probabil- 
ities of detecting any intervention effect are nearly identical to the 
probabilities of detecting an Intervention effect in the absence of one (Type I 
error). In other words, the stochastic fluctuations of a non-stationary series 
insure that any non-zero intervention is detectable with high probability. 
When a series is identified as non-stationary in level when it is non-stationary 
in level and slope, the results are that nominal power is an overestimate of 
the actual power for white noise cases, and that the actual power is nearly 
identical to the Type I error rate under a true null. As in the Type I error 
cases, the various models and their parameter values are more important in their 



influence on power. 

In situations where d is overestimated, the result is to grossly overestimate 
the power. Overdif f erencing a series removes the intervention effect from all 
but a few points (depending on the degree of over-differencing), and reduces 
power to the level of Type I error under a ttue null. ' 

Once again, the combination raisidentif ication cases reflect the importance 
of the d^ mis-specification. When a c[ = 1 process is underestimated (assumed 
d 3 0) , the power to detect a non-zero intervention effect is at least 0.80, 
the nominal power greatly underestimating the actual power. When a d_ = 2 
process is underestimated (assumed d_ = 1) and in those cases where d is over- 
estimated, the nominal power overestimates the actual power, consistent with 
the effects of the mis-specifications considered alone. In almost all cases 
the power is approximately the same as the probability of a Type I error under 
a true null hypothesis. Power results for the combined models are not the sum 
of the component power results. 

In conclusion, it is obvious that the proper identification of d^ is the 
highest priority of model identification: an error at this stage produces 
severe departures from nominal probabilities cf significance testing. Proper 
identification of d_ is a necessary but not sufficient condition for the researcher 
to operate at nominal probability levels. Once d is properly identified, other 
misidentif ications are possible, creating mildly discrepant to severe departures 
from the nominal probability values depending upon the particular underfit or 
misfit. 

Some general conclusions are warranted based upon the frequency of observed 
models in the socia 1 sciences . Since second differencing is rarely encountered 
(less than seven percent of the time), these statements apply to series where 

, 9 



the difference parameter is zero or one. 

In general) underdiff erencing places the researcher In the position of 
greatly underestimating powqr and operating very liberally with respect to 
Type I error; overdif ferencing produces situations where the researcher over- 
estimates power and operates conservatively with respect to Type I error. 
Misidentif ications where d is properly identified lead to either conservative 
or liberal situations, depending on the values of the model parameters. 

The researcher has two courses of action to minimize the chances of a 
model misidentif ication. The first is to select or generate time-series with 
a large number of observations (50 pre-I^ and post-I^ as a minimum). With a 
smaller number of points it is difficult to determine the degree of differencing 
required to attain s tationarity, since a "wandering'' over the short run may be 
either highly autocorrelated stochastic fluctuations of a stationary series or 
the "drifting" of a non-stationary series. 

In addition to identifying large series, the rei3earcher should always 
entertain more than one modil formulation and examine the residuals for the 
presence of autocorrelation. A statistical test is available (Box and Jenkins, 
1970, p. 503) and has been incorporated into a computer program developed by 
Bower, Padia and Glass (1974). 

Certainly, the researcher can minimize the possibility of model misidenti* 
fication and the severe accompanying effects on error probability statements 
by followingi* the above precautions and through a knowledge of the data. 
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